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Abstract 

Dynamic distributed storage algorithms such as DynaStore, Reconfigurable Paxos, RAMBO, and 
RDS, do not ensure liveness (wait-freedom) in asynchronous runs with infinitely many reconfigura¬ 
tions. We prove that this is inherent for asynchronous dynamic storage algorithms, including ones 
that use Q or o S oracles. Our result holds even if only one process may fail, provided that machines 
that were successfully removed from the system’s configuration may be switched off by an admin¬ 
istrator. Intuitively, the impossibility relies on the fact that a correct process can be suspected to 
have failed at any time, i.e., its failure is indistinguishable to other processes from slow delivery of 
its messages, and so the system should be able to reconfigure without waiting for this process to 
complete its pending operations. 

To circumvent this result, we define a dynamic eventually perfect failure detector, and present 
an algorithm that uses it to emulate wait-free dynamic atomic storage (with no restrictions on 
reconfiguration rate). Together, our results thus draw a sharp line between oracles like f l and o S, 
which allow some correct process to continue to be suspected forever, and a dynamic eventually 
perfect one, which does not. 



1 Introduction 


Many works in the last decade have dealt with the emulation of reliable storage via message passing in 
dynamic systems 

pn mm ei ei ei eh si. The motivation behind such systems is to allow the current 
configuration of participating processes to be changed; once a process is removed from the current 
configuration, a system administrator may shut it down, and the storage algorithm can no longer rely 
on it in order to ensure progress. We elaborate more on the requirement from dynamic storage in 
Section [2] To the best of our knowledge, no previous dynamic storage solution ensures completion of 
all operations in asynchronous runs with unrestricted reconfigurations, as detailed in Section [3} 

In Section [4j we show that this limitation is inherent as long as even one process that was not 
removed from the current configuration can fail. Specifically, we show that even a safe register emulation 
cannot guarantee liveness for all operations (i.e., wait-freedom) in asynchronous runs with unrestricted 
reconfigurations. The key to the impossibility proof is that, in asynchronous models, a slow correct 
process whose messages are delayed can be suspected to be faulty by all other processes, i.e., the two 
scenarios are indistinguishable and so the system should be able to reconfigure without allowing it to 
complete its operations. Our result holds even if processes are equipped with oracles like 0 or o S, which 
allow them to chose a leader and solve consensus in every configuration, yet some correct process can 
continue to be suspected forever. 

On the other hand, with a failure detector that guarantees a time after which correct processes are 
no longer suspected, our proof does not hold. Indeed, in Section [5j we define a dynamic version of 
the eventually perfect failure detector o P |2j, which we call oP D , and use it to implement a helping 
mechanism in order to circumvent the impossibility result. We present an algorithm, based on state 
machine replication, that emulates a wait-free atomic dynamic multi-writer, multi-reader (MWMR) 
register, and ensures liveness with unrestricted reconfigurations. Unlike fl-based reconhgurable state 
machine replication mm, our implementation ensures completion of all operations. 

Together, our results pinpoint the property required from an oracle failure detector for supporting 
wait-free dynamic storage. 

2 Model and Dynamic Storage Problem Definition 

2.1 Preliminaries 

We consider an asynchronous message passing system consisting of an infinite set of processes II. Each 
pair of processes is connected by a communication link. Processes may fail by crashing subject to 
restrictions on the number of failures given below. A service exposes a set of operations to its clients. 

An algorithm A defines the behaviors of processes as deterministic state machines, where state 
transitions are associated with actions, such as send/receive messages, operation invoke/response, and 
process failures. A global state is mapping to states from system components, i.e., processes and links. 
An initial global state is one where all processes are in initial states, and all links are empty. A run 
of algorithm A is a (finite or infinite) alternating sequence of global states and actions, beginning with 
some initial global state, such that state transitions occur according to A. We use the notion of time t 
during a run r to refer to the global state incurred following the t th action in r. A run fragment is a 
continuous subsequence of a run. An operation invoked before time t in run r is complete at time t if 
its response event occurs before time t in r; otherwise it is pending at time t. We assume that runs are 
well-formed [9], in that each process’s first action is an invocation of some operation, and a process does 
not invoke an operation before receiving a response to its last invoked one. We say that operation opi 
precedes operation opj in a run r, if opf s response occurs before opj's invocation in r, and operations 
opi and opj are concurrent in run r, if opt does not precede opj and opj does not precede opt in r. A 
sequential run is one with no concurrent operations. Two runs are equivalent if every process performs 
the same sequence of invoke and response actions in both (with the same return values). 
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2.2 Dynamic register service 

We consider a dynamic MWMR register service |Tj, which stores a value v from a domain V, and offers 
an interface for invoking read, write, and reconfiguration operations. Initially, the register holds some 
initial value vo G V. We define Changes to be the set { remove, add} x II, and call any subset of Changes 
a set of changes. For example, {(add,pfi), (remove,pf)} is a set of changes. A reconfig operation takes 
as a parameter a set of changes and returns “ok”. We say that a change w G Changes completes before 
time t in a run r , if some reconfig(c) completes in r before time t with w G c. We define Po C II to be 
the set of initial processes and say, by convention, that reconfig ({(add, p)\p G Po}) completes at time 0. 
We assume that Po is fixed and known to all. 

Notation For every subset w of Changes, the removal set of w, denoted w.remove, is 
{pi\(remove,pf) G u;}; the join set of w, denoted w.join, is {pi\(add,pi) G ie}; and the membership of 
w, denoted w.members, is w.join \ w.remove. For a time f in a run r, we denote by V(t) the union 
of all sets q s.t. reconfig(q) completes before time t in r. A configuration is a finite set of processes, 
and the current configuration at time t is V(t) .membership. We define P(t) to be the set of pending 
changes at time t in run r, i.e., the set of all changes included in pending reconfig operations, and we 
denote by F(t) the set of processes that have failed before time t in r, initially, P(0) = {}. For a series 
of sets S(t), we define S(*) = 

Correct processes and fairness A process p is correct if p G V(*).join\ F(*). A run r is 
fair if every enabled action by a correct process eventually occurs, and every message sent by a correct 
process pi to a correct process pj is eventually received at pj. A process p is active if p is correct and 
p fL P(*).remove. For simplicity, we assume that a process that has been removed is not added again. 

Service specification Atomicity, also called linearizabilty [9j, requires that for every run, after 
adding some response actions and then removing invocations that have no response, there exists an 
equivalent sequential run that conforms with the operation precedence relation, and satisfies the service’s 
sequential specification. The sequential specification for the register service is as follows: A read returns 
the latest written value or vq if none was written. 

Lamport |10j defines a safe single-writer register. Here, we generalize the definition to multi-writer 
registers in a weak way in order to strengthen the impossibility result: An MWMR register is safe, if 
in every sequential run r every read rd in r returns the register’s value when the read was invoked. 

A wait-free service guarantees that every active process’s operation completes, regardless of the 
actions of other processes. A wait-free dynamic atomic storage is a dynamic storage service that satisfies 
atomicity and wait-freedom, and a wait-free dynamic safe storage is one that satisfies safety and wait- 
freedom. 

Fault tolerance We now specify conditions on when processes are allowed to fail. First, we 
allow processes that are no longer part of the current configuration’s membership to be safely switched 
off. To capture this property, we say that a model is reconfigurable if at any time f in a run r, any 
process in V(t).remove can be in F(t). In other words, an adversary is allowed to crash any removed 
process. For our lower bound in Section[4j we define in addition the minimal failure condition: whenever 
( V(t).members U P(t).join ) n F(t) = {}, at least one process from V (t). member ship U P(t) can fail. 
In other words, whenever no unremoved process is faulty, the adversary is allowed to fail at least one 
unremoved process. 

The above two conditions strengthen the adversary, allowing it to fail processes in some scenarios. 
For our algorithm in Section [5j we need to also restrict the adversary, so as not to crash too many 
processes: We say that a model allows minority failures if at all times t in r, fewer than | V(t).members \ 
P(t).remove |/2 processes out of V(t).members U P(t).join are in F(t). 

Notice that whenever \V(t).members \ P(t).remove | > 3, the minority failure condition allows 
minimal failure. 

Suspicions A fundamental property of an asynchronous system is that failures cannot be accu¬ 
rately detected, and as long as processes can fail, a correct process can be suspected, in the sense that 
its failure is indistinguishable to other processes from slow delivery of its messages. To capture this 
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property in other models (e.g., ones with oracles), we define the following: 

A process p can be suspected at time t if (a) p has not failed before time t, (b) p can fail at time t 
according to the failure model; and (c) for any t' > t, every run fragment lasting from time t to time t' 
where p fails at time t is indistinguishable to all other processes from a run fragment where p is correct 
but all of its messages are delayed from time t to time t'. 

Failure detectors like oS or Q in a given configuration guarantee that there is eventually a time t 
s.t. there is one process in the configuration that cannot be suspected after time t. Nevertheless, other 
process may continue to be suspected forever: 

Observation 1. Consider an asynchronous model where processes are equipped with Q or oS in every 
configuration. Then in every run, there is some process that can be suspected at any time when its 
failure is allowed. 

Our impossibility result shows that wait-free dynamic storage emulation is impossible in such models 
even if only reconfigurability and minimal failure are required. 

3 Related Work 

Previous works on asynchronous dynamic storage assume either weak failure detectors like oS and 
Q [TUEi E;, 2} [IT), i3], or none at all mm- Therefore, they are all subject to our impossibility result in 
one way or another, as we now explain. 

Our minority and reconfigurability failure conditions are based on DynaStore ’s HUH] failure model, 
with the difference that we distinguish between removed processes and failed ones, and thus allow more 
failures. In addition, as long as there are at least three members in each current or pending configuration, 
minimal failures are allowed and so DynaStore is subject to our impossibility, and indeed, guarantees 
liveness under the assumption that number of reconfigurations is finite. 

RAMBO J8[ Uj and RDS [5] use on consensus to agree on reconfigurations, while read and write 
operations are asynchronous. They only discusses liveness and fault tolerance in synchronous runs with 
bounded churn and no guarantee on reconfigurability [8]. A similar liveness condition based on churn 
are used in mm- Therefore, these algorithm do not contradict our impossibility result. 

Reconhgurable Paxos variants mm provide dynamic state machine replication, and in turn imple¬ 
ment dynamic atomic storage. These works are subject to our impossibility result because they assume 
0 (a leader) in every configuration. A configuration may be changed, and accordingly a leader may 
be removed (and then fail) before a process p (with a pending operation) is able to communicate with 
it. Though a new leader is elected by D in the ensuing configuration, this scenario may repeat itself 
indefinitely. In Section [5j we augment state-machine replication with helping based on a stronger failure 
detector in order to avoid such scenarios. 

A related impossibility proof [2] shows that liveness is impossible to achieve with failures of more 
than a minority in the current configuration, which in some sense suggests that our minority failure 
condition is tight. 

4 Impossibility of Wait-Free Dynamic Safe Storage 

In this section we prove that there is no implementation of wait-free dynamic safe storage in a recon- 
figurable model that allows minimal failures and a correct process may be suspected forever. 

Theorem 1. Consider an asynchronous model allowing reconfigurability and minimal failures, where 
some correct process can be suspected at any time when its failure is allowed by the model. Then there 
is no algorithm that emulates a wait-free dynamic safe storage. 

Proof (Theorem [Tp. Assume by contradiction that such an algorithm A exists. We prove two lemmas 
about A. 
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Lemma 1.1. Consider a run r of A ending at time t, and some process pi that can he suspected at 
time t. Consider an active process pj in r that invokes operation op at time t. Then there exists an 
extension of r where (1) op completes at some time t' > t,(2) no process receives a message from 
Pi between t and t', and (3) pi does not fail. 

Proof (Lemma m Consider a run fragment that begins at time t, in which pi fails at time t and 
all of its in transit messages are lost. By wait-freedom, op eventually completes at some time t'. 
Since pi fails at time t and all its outstanding messages are lost, in the run fragment o\ starting 
from the global state at time t and ending when op is complete, no process receives any message 
from pi. Now let 02 be another run fragment lasting from time t to time t !, in which pi does not 
fail, but all of its messages are delayed. Recall that pi can be suspected at time t, therefore, 01 and 
(J 2 are indistinguishable to all processes except pi. Thus, op returns also in 02 • 

□ Lemma \l.l\ 


Lemma 1.2. Consider a sequential run r of A ending at time t, where some correct process pi can 
be suspected at any time in r and there is some active process pj / pi in r. Assume that no process 
invokes write(v 1 ) for some v\ / vq in r. If we extend r so that pi invokes w = write(y 1 ) at time 
t, and w completes at some time t! > t, then in the run fragment between t and t!, some process 
Pk / Pi receives a message sent by pi. 


Proof (Lemma 1.2). Assume by way of contradiction that w completes at some point t ', and in the 
run fragment between t and t! no process p^ 7 ^ pi receives a message sent by pi. Consider some 
other run r' that is identical to r until time t' except that pi does not invoke w at time t. Now 
assume that process pj invokes a read operation rd at time t' in r'. By the assumption, pi can be 
suspected at t'. Therefore, by Lemma there is a run fragment a of r' beginning at time t 1 , 
where rd completes at some time t", and no process receives a message from pi between t! and t". 
Since no other process invokes write(v 1 ) in r', rd returns some V 2 7 = v\. Now notice that all global 
states from time t to time t' in r and r' are indistinguishable to all processes except p , : . Thus, we 
can continue the run r with an invocation of read operation rd' by pj at time t !, and appending a 
to it. Operation rd! hence, completes and returns V 2 - A contradiction to safety. 

□ LemmaTM 


To prove the theorem, we construct an infinite fair run r in which a write operation of an active 
process never completes, in contradiction to wait-freedom. An illustration of the run for n = 4 is 
presented in Figure [l] 

Consider some initial global state Co, s.t. -P(O) = {} and V(0).members = {pi-..p n }- By the 
assumption, there is some process p that can be suspected at any time t when its failure is allowed. 
Assume w.l.o.g. that this process is p±, and let it invoke write operation w at time 0. Let t\ = 0. Now 
repeatedly do the following: 

Let process p n invoke reconfig(q) where q = {( add,pj)\n + 1 < j < 2n — 2 } at time t\. Since 
{V{t\).members U P(t\).join) n F(t\) = {}, p\ can fail according to the minimal failures condition, and 
therefore, by our assumption, can be suspected at time t\. So by Lemma we can extend r with 
a run fragment cii ending at some time t 2 when reconfig(q) completes, no process pj / p\ receives a 
message from p\ in < 7 i, and p\ does not fail. 

Then, at time t 2 , p n invokes reconfig(q') , where q’ = {(remove,pj )|2 < j < n — 1}. Again, 
(V(t 2 ).members U P(t 2 ).join ) fl F(t 2 ) = {}, and therefore, by our assumptions p\ can be suspected 
at time t 2 • And again, by Lemma [13 we can extend r with a run fragment (J 2 ending at some time £ 3 , 
when reconfig(q) completes, no process pj / p\ receives a message from p\ in 02 , and p\ does not fail. 

Recall that we assume a reconfigurable model, so all the processes in V(t^).remove can be now 
added to Ffo). Therefore, let the process in {pj | 2 < j < n — 1} fail at time £ 3 , and notice that the 
fairness condition does not mandate that they receive messages from p\. Next, allow p\ to perform all 
its enabled actions till some time £ 4 . 
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Now notice that at £ 4 , \V(t 4 ). members] = n, P(t±) = {}, and (V (£ 4 ) .members U P{t/f).join) n 
F(t/f) = {}. We can rename the processes in V(t/f).members (except p\) so that the process that 
performed the remove and add operations becomes p 2 , and all other get names in the range P 3 .. ,p n . 
We can then repeat the construction above. By doing so infinitely many times, we get an infinite run 
r in which p\ is active and no process ever receives a message from p\. However, all of pi’s enabled 
actions eventually occur. Since no process except p\ is correct in r, the run is fair. In addition, since 
(V(t).members U P(t).join ) n F(t) = {} for all t in r, by the minimal failures condition, p\ can fail at 
any time t in r. Hence, by the theorem’s assumption, can be suspected at any time t in r. Therefore, 
by Lemma 1.2 w does not complete in r, and we get a violation of wait-freedom. 

■ Theorem Q] 



P; 

Ps 


Figure 1: Illustration of the infinite run for n = 4. 

5 Oracle-Based Dynamic Atomic Storage 

We present an algorithm that circumvents the impossibility result of Section [4] using a failure detector 
In this section we a assume reconfigurable model with the minority failure condition. In Section 5.1 


we define a dynamic eventually perfect failure detector. In Section 5.2 we describe an algorithm, based 
on dynamic state machine replication, that uses the failure detector to implement a dynamic atomic 
MWMR register. The algorithm’s correctness is proven in Appendix |A| 


5.1 Dynamic failure detector 

Since the set of processes is potentially infinite, we cannot have the failure detector report the status 
of all processes as static failure detectors typically do. Dynamic failure detectors addressing this issue 
have been defined in previous works, either providing a set of processes that have been excluded from or 
included into the group [ 12 ], or assuming that there is eventually a fixed set of participating processes 0 . 

In our model, we do not assume that there is eventually a fixed set of participating processes, as 
the number of reconfigurations can be infinite. And we do not want the failure detector to answer with 
a list of processes, because in dynamic systems, this gives additional information about participating 
processes that could have been unknown to the inquiring process. Instead, our dynamic failure detector 
is queried separately about each process. For each query, it answers either fail or ok. It can be wrong 
for an unbounded period, but for each process, it eventually returns a correct answer. 

Formally, a dynamic eventually perfect failure detector, OP D , satisfies two properties: 

Strong Completeness: For each process pi that fails at time ti, there is a time t > U s.t. the failure 
detector answers fail to every query about pi after time t. 

Eventual Strong Accuracy: There exists a time t, called the stabilization time , s.t. the failure 
detector answers ok to every query at time t' > t about a correct process that was added before 
time t'. 
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5.2 Dynamic storage algorithm 
5.2.1 Algorithm overview 

State machine emulation of register We use a state machine sm to emulate a wait-free atomic 
dynamic register, DynaReg. Every process has a local replica of sm, and we use consensus |13j to agree 
on sm’s state transitions. Notice that each process is equipped with a failure detector FD of class OP D , 
so consensus is solvable under the assumption of a correct majority in a given configuration. 

Each consensus runs in a given configuration c, exposes a propose operation, and responds with 
decide, satisfies the following properties: By Uniform Agreement, every two decisions are the same. By 
Validity, every decision was previously proposed by one of the processes in c. By Termination, if a 
majority of c is correct, then eventually every correct processes in c decides. We further assume that a 
consensus instance does not decide until a majority of the members of the configuration propose in it. 

The sm (presented in lines [l]j4] in Algorithm [I]) keeps track of dynaReg’s value in a variable val, and 
the configuration in a variable eng, containing both a list of processes, cng.mem, and a set of removed 
processes, cng.rem. Write operations change val, and reconfig operations change eng. A consensus 
decision may bundle a number of operations to execute as a single state transition of sm. The number 
of state transitions executed by sm is stored in the variable ts. Finally, the array lastOps maps every 
process p in cng.mem to the sequence number (based on p’s local count) of p’s last operation that was 
performed on the emulated DynaReg together with its result. 

Each process partakes in at most one consensus at a time; this consensus is associated with timestamp 
sm.ts and runs in S7n.cng.mem. In every consensus, up to \sm.c7ig.mem\ ordered operations on the 
emulated DynaReg are agreed upon, and sm’s state changes according to the agreed operations. A 
process’s sm may change either when consensus decides or when the process receives a newer sm from 
another process, in which case it skips forward. So sm goes through the same states in all the processes, 
except when skipping forward. Thus, for every two processes Pk,Pi, if sink-ts = smi.ts, then sink = smi. 
(A subscript i indicates the variable is of process p t .) 

Helping The problematic scenario in the impossibility proof of Section [4] occurs because of endless 
reconfigurations, where a slow process is never able to communicate with members of its configuration 
before they are removed. In order to circumvent this problem, we use the FD to implement a helping 
mechanism. When proposing an operation, process pi tries to help other processes in two ways: first, it 
helps them complete operations they may have successfully proposed in previous rounds but have not 
learned about their outcome; and second, it proposes their new operations. First, it sends its sm to all 
other processes in smi.eng.mem, and waits for each to reply with its latest invoked operation. Then pi 
proposes all the operations together. Processes may fail or be removed, so pi cannot wait for answers 
forever. To this end, we use the FD. For every process in smi.cng.mem that has not been removed, pi 
repeatedly inquires FD and waits either for a reply from the process or for an answer from the FD that 
the process has failed. Notice that the strong completeness property guarantees that pi will eventually 
continue, and strong accuracy guarantees that every slow active process will eventually receive help in 
case of endless reconfigurations. 

Nevertheless, if the number of reconfigurations is finite, it may be the case that some slow process 
is not familiar with any of the correct members in the current configuration, and no other process 
performs an operation (hence, no process is helping). To ensure progress in such cases, every correct 
process periodically sends its sm to all processes in its sm.cng.mem 

State survival Before the reconfig operation can complete, the new sm needs to propagate to a 
majority of the new configuration, in order to ensure its survival. Therefore, after executing the state 
transition, pi sends smi to smi.eng members and waits until it either receives acknowledgements from a 
majority or learns of a newer sm. Notice that in latter the case, consensus in snii.cng.mem has decided, 
meaning that at least a majority of snii.cng.mem have participated in it, and so have learned of it. 

Flow example The algorithm flow is illustrated in Figure [2j In this example, a slow process p 2 
invokes operation op 2 i before the FD’s stabilization time, ST. Process p\ invokes operation opu = 
{<add,p 3 ) after ST. It first sends helpRequest to p 2 and waits for it to reply with helpReply. Then it 


6 


proposes op 2 i and opu in a consensus. When decide occurs, p\ updates its sm , sends it to all processes, 
and waits for majority. Then opu returns and p\ fails before P 2 receives its update message. Next, p^ 
invokes a reconfig operation, but this time when p 2 receives helpRequest with the up-to-date sm from 
P 3 , it notices that its operation has been performed, and op 2 \ returns. 



Figure 2: Flow illustration: process P 2 is slow. After stabilization time, process p\ helps it by proposing 
its operation. Once P 2 S operation is decided, it is reflected in every up-to-date sm. Therefore, even if 
pi fails before informing p 2 - P 2 receives from the next process that performs an operation, namely, p%, 
an sm that reflects its operation, and thus returns. Line arrows represent messages, and block arrows 
represent operation or consensus invocations and responses. 
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5.2.2 Detailed description 

The data structure of process pi is given in Algorithm]!] First, snii, is described above. Integer opNum.i 
holds the sequence number of pi s current operation; opsi is a set that contains operations that need to 
be completed for helping; the flag pendi is a boolean that indicates whether or not pi is participating 
in an ongoing consensus; and myOpi is the latest operation invoked at p^. 

The algorithm of process pi is presented in Algorithms [2] and [3j We execute every event handler, 
(operation invocation, message receiving, and consensus decision), atomically excluding wait instruc¬ 
tions; that is, other event handlers may run after the handler completes or during a wait (lines 7 | 10| 19 
in Algorithm^. The algorithm runs in two phases. The first, gather , is described in Algorithm [2] lines 
and in Algorithm [3] lines [24 29 


30 32] Process pi first increases its operation number opNurrii, 


writes op together with opNunn to the set of operations opsi, and sets myOpi to be op. Then it sends 
(“helpRequest”,...) to every member of A = srrii.cng.mem (line [6]), and waits for each process in A 
that is not suspected by the FD or removed to reply with (“helpReply”,...). Notice that srrii may 
change during the wait because messages are handled, and pi may learn of processes that have been 
removed. 

When (“helpRequest”, num, sm) is received by process pj pi, if the received sm is newer than srrij, 
then process pj adopts sm and abandons any previous consensus. Either way, pj sends (“helpReply”,...) 
with its current operation myOpj in return. 

Upon receiving (“helpReply”, opNurrii, op, num) that corresponds to the current operation number 
opNumi , process pi adds the received operation op , its number num, and the identity of the sender to 
the set opsi. 

At the end of this phase, process pi holds a set of operations, including its own, that it tries to 
agree on in the second phase. Note that pi can participate in at most one consensus per timestamp, 
and its propose might end up not being the decided one, in which case it may need to propose the 
same operations again. Process pi completes op when it discovers that op has been performed in smi, 
whether by itself or by another process. 

The second phase appears in Algorithm [2] lines |8]|21[ and in Algorithm [3] lines [l 16 17 23 In line 
[HJ pi checks if its operation has not been completed yet. In lines [9] to it waits until it does not 
participate in any ongoing consensus ( pendi=false) or some other process helps it complete op. Recall 
that during a wait, other events can be handled. So if a message with an up-to-date sm is received 
during the wait, pi adopts the sm. In case op has been completed in sm, pi exits the main while (line 
11). Otherwise, pi waits until either it does not participate in any ongoing consensus. This can be the 
case if (1) pi has not proposed yet, (2) a message with a newer sm was received and a previous consensus 
was subsequently abandoned, or (3) a decide event has been handled. In all cases, pt marks that it now 
participates in consensus in line [121 prepares a new request Req with the operations in opsi that have 


not been performed yet in smi hi line 19, proposes Req in the consensus associated with smi.ts, and 
sends (“propose”,...) to all the members of smi.eng.mem. 

When (“propose”, sm, Req.. .) is received by process pj pi, if the received sm is more updated 
than srrij, then process pj adopts sm, abandons any previous consensus, proposes Req in the consensus 
associated with sm.ts, and forwards the message to all other members of smj .eng.mem. The same is 
done if sm is identical to smj and pj has not proposed yet in the consensus associated with snij.ts. 
Otherwise, pj ignores the message. 

The event decidefism.cng, smi.ts, Req ) indicates a decision in the consensus associated with smi.ts. 
When this occurs, pi performs all the operations in Req and changes smi s state. It sets the value of the 
emulated DynaReg, smi.value, to be the value of the write operation of the process with the lowest id, 
and updates smi.eng according to the reconfig operations. In addition, for every ( pj,op,num ) 6 Req, 
Pi writes to smi.lastOps[j ], num and op’s response, which is “ok” in case of a write or a reconfig, and 
smi.value in case of a read. Next, pi increases smi.ts and sets pendi to false, indicating that it no longer 
participates in any ongoing consensus. 

Finally, after op is performed, pi exits the main while. If op is not a reconfig operation, then pi 
















returns the result, which is stored in smi.last.Ops[i\.res. Otherwise, before returning, pi has to be 
sure that a majority of srrii.cng.mem> receives smi. It sends (“update”, sm, ...} to all the processes in 
smi.cng.mem and waits for (“ACK”,...} from a majority of them. Notice that it may be the case that 
there is no such correct majority due to later reconfigurations and failures, so, pt stops waiting when a 
more updated sm is received, which implies that a majority of smi.cng .mem has already received sm>i 
(since a majority is needed in order to solve consensus). 

Upon receiving (“update”, sm,num) with a new sm from process pi, process pj adopts sm and 
abandons any previous consensus. In addition, if num. ^_L, pj sends (“ACK”, num) to pi (Algorithm [3] 
lines 33 38). 

Beyond handling operations, in order to ensure progress in case no operations are invoked from some 
point on, every correct process periodically sends (“update”, sm, _L) to all processes in its sm.cng.m.em 
(Algorithm [2] line [23]) . 

In Appendix [A] we prove that the algorithm satisfies atomicity and wait-freedom. 


Algorithm 1 Data structure of process pi 
1: smi.ts £ N, initially 0 
2: srrii.value £ VU { _L}, initially _L 
3: smi.cng = {mem,rem), 

where mem,rem C II, initially (Po,{}), where Pg C II 
4: smi.lastOps is a vector of size \smi.cng.mem\, 

where \/pj £ smi.cng.mem , smi.lastOps[j] = (num, res), 
where num £ N, res £ V U {_L,“ok”}), initially (0, _L) 

5: pendi £ {true,false}, initially false 

6: opNumi £ N, initially 0 

7: opSi C II x operation x N , initially {} 

where operation = {( RD ), (WR, value), ( REC, changes), _L}, 
where value £ V, changes C {add, remove} x II, and type can be RD,WR, REC 
8: myOpi £ operation, initially _L 


Algorithm 2 Process pf s algorithm: performing operations 
1: upon invoke operation(op) do 

2: opNumi £- opNumi + 1 > phase 1: gather 

3: opSi £- {(pi, op, opNumi)} 

4: myOpi ■£- op 

5: A <— smi.cng.mem 

6: for all p £ A send (“helpRequest”, opNumi, smf) to p 

7: for all p £ A wait for (“helpReply”, opNumi,...) from p or p is suspected or p £ smi.cng.rem 

8: while smi.lastOps[i\.num ^ opNumi > phase 2: agree&perform 

9: while (pendi) 

10: wait until ->pendi or smi.lastOps[i\.num = opNumi 

11: if sm.i.lastOps[i\.num = opNumi then goto line [l6] 

12: pendi <— true 

13: Req ■£- {(pj,op,num) £ opSi \ num > smi.lastOps[j].num} 

14: propose(smi.cng, smi.ts, Req) 

15: for all p £ smi.cng.mem send (“propose”, smi, Req) to p 

16: if op.type = REC 

17: ts ■£- smi.ts 

18: for all p £ smi.cng.mem send (“update”, smi, opNumi) to p 

19: wait for (“ ACK” , opNumi) from majority of smi.cng.mem or smi.ts > ts 

20: return smi.lastOps[i\.res 

21: end 

22: periodically: 

23: for all p £ smi.cng.mem send (“update”, smi, -L) to p 
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Algorithm 3 Process pf s algorithm: event handlers 

1: upon decide.i(smi.eng, sm.i.ts, Req) do 
2 : W 4— {(p, value)\(p, (WR, value), num) € Req} 

3: if W ± {} 

4: snii.value 4— value associated with smallest p in W 

5: for all (p, op, num) € Req 

6: if op.type = WR 

7: sm.i.lastOps[j] 4— (num, “ok”) 

8: else if op.type = RD 

9: smi.lastOps[j] 4— (num, smi.value) 

10: else 

11: smi.cng.mem 4— {p\(p £ smi.cng.mem & (remove,p) op.changes) || 

((add,p) € op.changes & p ef sm.i.cng.rem)} 

12 : smi.cng.rem 4— {p\(p € sm.cng.rem || (remove,p) £ Changes} 

13: sm.i.lastOps[j] 4— (num, “ok”) 

14: smi.ts 4— smi.ts + 1 

15: pendi 4— false 

16: end 

17: upon receiving ( “propose” , sm, Req) from pj do 

18: if ( sm.i.ts > sm.ts) or ( smi.ts = sm.ts & pendi = true) then return 

19: smi £- sm 

20 : pendi £~ true 

21: propose(smi..cng, smi.ts, Req) 

22: for all p £ smi.cng.mem send (“propose”, smi, Req) to p 

23: end 

24: upon receiving ( “helpRequest” , num., sm) from p,j do 
25: if sm.i.ts < sm.ts then 

26: smi t— sm. 

27: pendi <— false 

28: send (“helpReply” , num, myOpi, opNum.i) 

29: end 

30: upon receiving (“helpReply”, opNumi, op, num) from pj do 
31: opsi 4— opsi U ( Pj, op, num) 

32: end 

33: upon receiving ( “update” , sm, num) from pj do 
34: if sm.i.ts < sm.ts then 

35: smi £- sm. 

36: pendi false 

37: if num then send ( “ACK” , num) to pj 

38: end 


6 Conclusion 

We proved that in an asynchronous reconfigurable model allowing at least one failure, and no restriction 
on the number of reconfigurations, there is no emulation of dynamic wait-free storage. This is true even 
for safe storage, and even if processes are equipped with 0 or oS failure detectors, which allow them 
to solve consensus in every configuration. We further showed how to circumvent this result using a 
dynamic eventually perfect failure detector: we presented an algorithm that uses such a failure detector 
in order to emulate a wait-free dynamic atomic MWMR register. 

Our results thus draw a distinction between models where correct processes can be suspected at any 
time (as long as they may fail), and ones where false suspicions eventually cease. 
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A Correctness Proof 

Here we prove the correctness of our algorithm (Section [5]). 


A.l Atomicity 

Every operation is uniquely defined by the process that invoked it and its local number. During the proof 
we refer to operation op invoked by process pi with local number opNumi as the tuple (pi, op, opNurm), 
or simply as opNum.i. We begin the proof with three lemmas that link completed operation to sm. 
states. 


Lemma 1.3. Consider operation op is invoked by some process pt in r with local number opNumi. If 
op returns in r at time t, then there is at least one request Req that contains (pi, op, opNunii) and has 
been chosen in a consensus in r before time t. 


Proof. Operation op cannot return until smi.lastOps[i].num = opNumi (line[8]or 10 in Algorithm [2]) . 
Processes update sm during a decide handler, or when newer sm is received. Easy to show by induction 
that some process pj writes opNumi to smi.lastOps[i\.num during a decide handler. According to the 
run of the decide handler, opNumi is written to sm.lastOps[i].num only if the chosen request in the 
corresponding consensus contains (pi, op, opNumi). 

□ 


Lemma 1.4. For every two processes Pi,Pj. Let t be a time in r in which neither pi or pj executing 
decide handler. Then at time t, if s.mi.ts = smj.ts, then smi = smj. 

Proof. We prove by induction on timestamps. Initially, all correct processes have the same sm with 
timestamp 0. Now consider timestamp TS, and assume that for every two processes Pi,Pj at any time 
not during the execution of decide handlers, if smi.ts = smj.ts = TS, then smi = smj. Processes 
increase their sm.ts to TS + 1 either at the end of a decide handler associated with TS or when they 
receive a message with sm s.t. sm.ts = TS + 1. By the agreement property of consensus and by the 
determinism of the algorithm, all the processes that perform the decide handler associated with TS, 
perform the same operations, and therefore move sm (at the end of the handler) to the same state. It 
is easy to show by induction that all the processes that receive a message with sm s.t. sm.ts = TS +1, 
receive the same sm. The lemma follows. 

□ 

Observation 2. For any two states sm±, sm 2 , and for any process pi in a run r, if smi.ts > sm 2 -ts, 
then smi.lastOps[i].num > sm 2 -lastOps[i\.num. 

Proof. Easy to show by induction. 

□ 


Lemma 1.5. Consider operation op invoked in r by some process pi with local number opNumi in r. 
Then op is part of at most one request that is chosen in a consensus in r. 


Proof. Assume by way of contradiction that op is part of more than one request that is chosen in 
a consensus in r. Now consider the earliest one, Req, and assume that it is chosen in a consensus 
associated with timestamp TS. At the end of the decide handler associated with timestamp TS, 
sm.lastOps[i].num = opNumi and it is increased to TS' + l. Thus, by Lemma 1.4 sm.lastOps[i\.num = 
opNumi holds for every sm s.t. sm..ts = TS + 1. Consider now the next request, Req\, that contains 
op, and is chosen in a consensus. Assume that this consensus associated with ts TS', and notice that 
TS' > TS. By the validity of consensus, this request is proposed by some process pj, when smj.ts is 
equal to TS'. By Observation [2j sm.lastOps[i\.num = opNumi holds for all sm s.t. sm.ts = TS', and 
therefore pj does not enter op to Reqi (line [l9] in Algorithm [2]) . A contradiction. 

□ 
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Based on the above lemmas, we can define, for each run r, a linearization oy, where operations are 
ordered as they are chosen for execution on sm’s in r. 

Definition 1. For a run r, we define the sequential run oy to be the sequence of operations decided in 
consensus instances in r, ordered by the order of the chosen requests they are part of in r. The order 
among operations that are part of the same chosen request is the following: first all writes, then all 
reads , and finally, all reconfig operations. Among each type, operations are ordered by the process ids 
of the processes that invoked them, from the highest to the lowest. 


Corollary 1. For every run r, the sequential execution oy is well defined. That is, oy contains every 
completed operation in r exactly once, and every invoked operation at most once. 

In order to prove atomicity it remains to show that (1) oy preserves r’s real time order; and (2) every 
read operation rd in r returns the value that was written by the last write operation that precedes rd 
in ay, or _L if there no such operation. 


Lemma 1.6. If operation opi returns before operation op 2 is invoked in r, then opi appears before op 2 
in oy. 


Proof. Operation op\ returns before operation op 2 is invoked in r. Therefore By Lemma 1.3 op\ is part 
of a request Req\ that is chosen in a consensus before op 2 is invoked, and thus op 2 cannot be part of 
Reqi or any other request that is chosen before Reqi. Hence op\ appears before op 2 in ay. 

□ 


Lemma 1.7. Consider read operation rd invoked by some process pi with local number opNum.i in r, 
which returns a value v. Then v is written by the last write operation that precedes rd in oy, or v =T 
if there is no such operation. 


Proof. By Lemmas 1.3 and 1.5, rd is part of exactly one request Reqi that is chosen in a consensus, 
associated with some timestamp TS. Thus sm.lastOps[i\ is set to {opNunii, sm.value) in the decide 
handler associated with TS, denote the value of sm.value at this point to be val. By Lemma 1.4 


sm.lastOps[i] = {opNumi,val) for all sm s.t. sm.ts = TS + 1. By Lemma 1.5, no process write to its 
sm.lastOps[i ] until rd returns, so smi.lastOps[i\ = (opNumi,val) when rd returns, and therefore rd 
returns val. Now consider three cases: 


• There is no write operation in Reqi or in any request that was chosen before Req\ in r. In 
this case, there is no write operation before rd in ay, and no process writes to sm.value before 
sm.lastOps[i\ is set to (opNum.i, sm.value), and therefore, rd returns _L as expected. 

• There is a write operation in Reqi in r. Consider the write operation w in Reqi that is invoked 
by the process with the lowest id, and assume its argument is v'. Notice that w is the last write 
that precedes rd in oy. By the code of the decide handler, sm.value equals v 1 at the time when 
sm.lastOps[i ] is set to (opNumi, sm.value). Therefore, rd returns v' , which is the value that is 
written by the last write operation that precedes it in oy. 


There is no write operation in Reqi, but there is a request that contains a write operation and is 
chosen before Reqi in r. Consider the last such request Req 2 , and consider the write operation 
w invoked by the process with the lowest id in Req 2 - Assume that re’s argument is v', and Req 2 
was chosen in a consensus associated with timestamp TS' (notice that TS' < TS). By the code 
of the decide handler and Lemma 1.4, in all the sm’s s.t. sm.ts = TS' + 1, the value of sm..value 
is v'. Now, since there is no write operation in any chosen request between Req 2 and Reqi in 
r, no process writes to sm.value s.t. TS' < sm.ts < TS. Hence, when sm.lastOps[i\ is set to 
( opNunii, sm.value), sm.value equals v', and therefore rd returns v'. The operation w is the last 
write operation that precedes rd in ay. Therefore rd returns the value that is written by the last 
write operation that precedes rd in ay. 
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□ 


Corollary 2. The algorithm of Sectionals atomic. 


A. 2 Liveness 


Consider operation opt invoked at time t by a correct process pi in run of r. Notice that r is a run with 
either infinitely or finitely many invocations. We show that, in both cases, if pi is active in r, then opi 
returns in r. 

We associate the addition or removal of process pj by a process pi with timestamp that equals snii.ts 
at the time when the operation returns. The addition of all processes in Pq is associated with timestamp 
0 . 

First, we consider runs with infinitely many invocations. In Lemma 1.8, we show that for every 


process p, every sm associated with a larger timestamp than p’s addition contains p in sm.cng.mem. 
In Observation [3j we show that in a run with infinitely many invocation, for every timestamp ts, there 
is a completed operation that has a bigger timestamp than ts at the time of the invocation. Moreover, 
after the stabilization time of the FD, operations must help all the slow active processes in order to 
complete. In Lemma 1.9 we use the observation to show that any operation invoked in a run with 


infinitely many invocations returns. 

Next, we consider runs with finitely many invocations. We show in Observation [6] that there is a 
correct majority in every up-to-date configuration, and in Lemma 1.10, we show that eventually all the 


active members of the last sm adopt it. Then, in Lemma 1.11| we show that every operation invoked 
by active process completes. Finally, in Theorem [2j we that the algorithm satisfies wait-freedom. 

Lemma 1.8. Assume the addition of pi is associated with timestamp TS in run r. If pi is active, then 
Pi E sm.cng.mem for every sm s.t. sm.ts >TS. 

Proof. The proof is by induction on sm.ts. Base: If pi E Pq, then pi E sm.cng.mem for all sm 
s.t. sm.ts = 0. Otherwise, (add,pf) is part of a request that is chosen in a consensus associated with 


timestamp TS' = TS— 1, and thus, by with Lemma 1.4, pi E sm.cng.mem for all sm s.t. sm.ts = TS'+l. 
Induction: Process pi is active, so no process invokes (remove, pi), and therefore, together with the 
validity of consensus, no chosen request contains (remove, pf). Hence, if p t E sm.cng.mem for sm with 
sm.ts = k, then pi E sm.cng.mem for every sm s.t. sm.ts = k + 1. 

□ 


Observation 3. Consider a run r of the algorithm with infinitely many invocations. Then for every 
time t and timestamp TS, there is a completed operation that is invoked after time t by a process with 
sm.ts > TS at the time of the invocation. 

Proof. Recall that r is well-formed. Therefore, there are infinitely many completed operations in r. Now 
notice that a process cannot invoke two operations with the same sm.ts. For every timestamp TS, at 
the time of the decision in the consensus associated with TS, there are finitely many correct processes. 
All processes whose addition is associated with timestamps bigger than TS, never have sm.ts < TS. 
Hence, a finite number of operations are invoked by processes with sm.ts < TS at the time of the 
invocation. And therefore, after every time t, there are completed operations that are invoked by a 
processes with sm.ts > TS at the time of the invocation. 


Lemma 1.9. Consider an operation opt invoked at time t by an active process pi in a run r with 
infinitely many invocations. Then opi completes in r. 

Proof. Assume by way of contradiction that pi is active and opi does not complete in r. Assume that 
Pi s addition is associated with timestamps TS. Consider a time t' > t after p t invoked op, and the 
FD has stabilized. By Observation [3j there is a completed operation opj in r, invoked by some process 
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Pj at a time t" > t' when srrij.ts > TS, which completion is associated with timestamp TS'. By 
Lemma 1.8, Pi £ srrij.cng.mem, at time t". Now by the algorithm and by the eventual strong accuracy 
property of the FD, pj proposes opj and opi in the same request, and continue to propose both of them 
until one is selected. Note that it is impossible for opi to be selected without opi since any process 
that helps pi after stabilization also helps p*. Hence, since opi completes, they are both performed in 
the same decide handler. The run is well-formed, so pi does not invoke operations that are associated 
with a higher num. Hence, following the time when opi is selected, for all sm s.t. sm.ts > TS', 
sm.lastOps[i\.num = opNurrii. Now, again by Observation [3j consider a completed operation opk in r, 
that is invoked by some process pk at time t'" after the stabilization time of the FD s.t. snik-ts > TS' 
at time t'". Operation opk cannot complete until pi receives pk s sm. Therefore, pi receives sm s.t. 
sm.ts > TS', and thus sm.lastOps[i\.num = opNumi. Therefore, p r learns that opi was performed, 
and opt completes. A contradiction. 

□ 


We now proceed to prove liveness in runs with finitely many invocations. 


Definition 2. For every run r of the algorithm, and for any point t in r, let TSt be the timestamp 
associated with the last consensus that made a decision in r before time t. Define sm t , at any point t 
in r, to be the sra’s state after the completion of the decide handler associated with timestamp TSt at 
any process. Recall that sm° is the initial state. 

Observation 4. For every run r of the algorithm, and for any point t in r, smf is unique. 


Proof. By Lemma 1.4 all the decide handlers associated with the same timestamp lead to the same 
sm. 

□ 


Observation 5. For every run r of the algorithm, and for any point t inr, V(t).members C sm'.cng.mem 
and smf .eng .mem V(t).remove = {}. 

Proof. Easy to show by induction. 

□ 


Observation 6. For every run r of the algorithm, and for any point t in r. There is a majority of 
sinf.cng.mem M s.t. M C [V(t).member sU P(t),join) \F(t). 

Proof. The observation follows from Observation [5] and the failure condition. 
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Observation 7. Consider a run r of the algorithm with finitely many invocations. Then there is a 
point t in r s.t. for every t' > t, sm t = sm t . Denote this sm to be sin. 

Lemma 1.10. Consider a run r of the algorithm with finitely many invocations. Then eventually for 
every active process pi £ sin.cng.mem, smi = sm. 


Proof. Recall that initially sm^.eng .mem = Pq, and for every process pt £ Pq, smi = smo. Therefore 
the lemma holds if sm = smo. Now assume that sin. ^ smo, and assume that sm.ts = TS. Consider 

Assume that 


prev • 


By Lemma 1.4 


sm prev is well defined. 


the sm s.t. sm.ts = TS — 1, denote it sm. 
the decision in the consensus associated with TS — 1, denote it con, was made at some time if. Now 
consider two possible cases. 

In first case, there is no reconfig operation that was chosen in con and completes in r. By Observation 
[5j V(t').members C sm prev .eng.mem and sm prev .eng.mem C V(t').remove = {} at time t', so by the 
Observation [6] there is a correct majority of sm prev .cng.mem. By our assumption on consensus, a 
majority of sm prev .eng.mem has to propose in order to made a decision. Therefore, since the majority 
interact, there is some active process in sm prev .eng.mem that decides in con, and moves its state to 
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sfn. Now recall that the processes periodically send update messages with their sm to all the process 
in their sm.eng.mem. Therefore, for every active process pi in sm prev .cng.mem, eventually sm* = sfn. 

In the second case, some reconfig operation that was chosen in con completes. Notice that its com¬ 
pletion must be associated with timestamp TS. Therefore, by the algorithm (lines 16 19 in Algorithm 
[2]), a majority of sfn.eng.mem receives sfn. By the failure condition, at least one of these processes is 
active. Hence, thanks to the periodic update messages, for every active process pt in sm prev .eng.mem, 
eventually smi = sfn. 

□ 


Lemma 1.11. Consider an operation opi invoked at time t by an active process pi in a run r with 
finitely many invocations. Then opi completes in r. 


Proof. By Lemma 1.8, pi £ sfn.cng.mem , and by Lemma 1.10, there is a point t in r s.t. sm,; = sm 
for all t > t'. Assume by way of contradiction that opi does not complete in r. Therefore, opi is either 
stuck in one of its waits or continuously iterates in a while loop. In each case, we show a contradiction. 
Denote by con the consensus associated with timestamp sfn.ts. By definition of sfn, no decision is made 


m con m r. 


• Operation opi waits in line [T] (Algorithm [ 2 ]) forever. Notice that sfn.cng.rem contains all the 
process that were removed in r, so, after time t!, pi does not wait for a reply from a removed 
process. By the strong completeness property of FD, pi does not wait for faulty processes forever. 
A contradiction. 

• Operation opi remains in the while loop in line [ 9 ] (Algorithm [ 2 ]) forever. Notice that from time t' 
till pi proposes in con, pendi=false. Therefore, p% proposes in con in line [l4] (Algorithm [ 2 ]), and 
stays in the while after the propose. By Observation [6j there is a majority M of sfn.cng.mem s.t. 
M C V(t).members U P(t).join \ F(t). Therefore, by the termination of consensus, eventually a 
decision is made in con. A contradiction to the definition of sfn. 

• Operation opi remains in the while loop in line [8] (Algorithm [ 2 ]) forever. Since it does not remain 
in the while loop in line [9j opi proposes infinitely many times, and since each propose is made in a 
different consensus and pi can propose in a consensus beyond first one only once a decision is made 
in the previous one, infinitely many decisions are made in r. A contradiction to the definition of 
sfn. 


Operation opi waits in line |19| (Algorithm [ 2 ]) forever. Consider two cases. First, smi sfn when 
Pi performs line |18| (Algorithm [ 2 ]). In this case, pi continues at time t', when it adopts sfn, because 
smi.ts > ts hold at time t'. In the second case (smi = sfn when pi performs line 18), pi sends 
update message to all processes in sfn.cng.mem, and waits for a majority to reply. By Observation 
[6j there is a majority M of sfn.cng.mem , s.t. M C V(t).members U P(t).join \ F(t). Therefore, 
eventually pi receives replies from all the process in M, and thus continues. In both cases we have 
contradiction. 


Therefore, pi completes in r. 

a 


Theorem 2. The algorithm of Section^ implements a wait-free 


Proof. By Lemmas |1,9| and 1.11, every operation, invoked in r by 
Corollary [2j the algorithm is atomic. 


atomic dynamic storage. 

an active process, completes. And by 


□ 
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